The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.
Messaging is a communication model that is used to handle the complexity of communications between multiple entities in one or more computer systems. In the context of computer systems, the term “message” may refer to any item that includes data of any data type or format. For example, a database application for a Database Management System (DBMS) may submit a request for data in the form of a message to a database server and the request is stored in a message queue of the DBMS. Furthermore, messages may be stored persistently, may be propagated between queues on different nodes of a distributed DBMS, and may be transmitted over a network.
As used herein, the terms “message queue” and “queue” refer to a message queue implemented in a volatile memory, such as a Random Access Memory (RAM). The volatile memory may be a shared volatile memory that is accessible by a plurality of processes. A message queue may also be used in conjunction with a storage space in non-volatile permanent store for storing messages from the queue, such as, for example, a database, a table in a database, a file system, or a file in a file system. Message queues used in conjunction with storage space in permanent store are typically used as a communication mechanism by information systems that require a high quality of messaging service, such as, for example, guaranteed delivery of messages even in the event of a information system crash or failure.
A “spill” process flushes messages from a message queue to a non-volatile permanent store, and is typically used to manage the amount of available space in the message queue. For example, the spill process addresses situations where a request is made to enqueue a new message into a message queue when the queue does not currently have sufficient available space to store the new message. In order to make room for the new message in the message queue, the spill process stores (“spills over”) one or more messages from the queue to the permanent store. Only the message headers of the spilled messages remain in the message queue in order to maintain the place of the spilled messages in the queue. A message header typically contains data indicating that its associated message is stored in the permanent store, and may also specify the location in the permanent store where the message is stored. When a message header is processed, the message associated with the header is retrieved from the permanent store. According to some implementations of a spill process, spilled messages are stored in a permanent store that is organized as a queue. Further, the spill process is usually transparent to entities that use the message queue, and such entities are usually unaware of whether the messages they are interested in have been stored in the permanent store.
The message queues referred to herein support a publish-and-subscribe communication mechanism, where message producers and message consumers may be decoupled from and independent of each other. An entity that produces a message is referred to as a “publisher.” An entity interested in messages in a message queue “subscribes” to the message queue and is referred to as a “subscriber” or a “consumer”. The “publisher” and “consumer” entities may be any process, device, software application, daemon, thread, fiber, or any other mechanism that is allocated computing resources and is executing in one or more computer systems. When a publisher “publishes”, or “enqueues”, messages to a message queue, the messages become available to the consumers who may “consume”, or “dequeue”, the messages that they have subscribed for from the message queue. Usually, a message is removed, or deleted, from the queue only after every consumer to which the message is targeted has consumed the message. If a message has not yet been consumed by all of its intended consumers, the message typically stays in the queue.
In some implementations of message queues that are used in conjunction with a permanent store, messages published in a message queue are delivered to all eligible consumers at least once. In these implementations, consumers of messages in the message queue are guaranteed delivery even in the event of failures, so long as a publisher is “repeatable.” A publisher is “repeatable” when it re-enqueues, in response to the occurrence of a particular event or a failure, all messages that (1) it published to the message queue before the event or failure and (2) have not yet been consumed by all consumers that have subscribed to these messages. The operation in which a publisher re-enqueues, after the occurrence of an event or a failure, messages that it previously has enqueued in the queue is referred to herein as a “replay” operation. An example of a repeatable publisher is an application, in a DBMS, that implements transactional replication, in which changes made to a database in one location must be replicated to one or more other databases in different locations.
In a system that has multiple publishers and multiple consumers, and in which messages may be transferred from some queues to other queues, the specific techniques used to manage messages in the system can have a significant impact on the performance in areas such as recoverability and memory usage. Therefore it is desirable to provide mechanisms for efficiently managing the publishers, the queues, the propagation of messages, and the resources involved in maintaining the queues.